<!DOCTYPE html>
<html lang="zh">
  <head>
    <meta charset="UTF-8" />
    <meta name="author" content="xmy6364">
    <meta name="description" content="想要成为大佬">
    <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=no" />

    <link rel="shortcut icon" href="/assets/favicon.ico" type="image/x-icon">
    <link rel="preconnect" href="https://fonts.gstatic.com">
    <link href="https://fonts.googleapis.com/css2?family=Noto+Serif+SC:wght@600&display=swap" rel="stylesheet">
    <link rel="stylesheet" href="https://cdn.bootcdn.net/ajax/libs/highlight.js/10.5.0/styles/default.min.css">
    <link rel="stylesheet" href="https://cdn.bootcdn.net/ajax/libs/highlight.js/10.5.0/styles/github-gist.min.css">
    <link rel="stylesheet" href="/assets/lib/css/font-awesome.min.css">
    <link rel="stylesheet" href="/assets/css/layout.css" />
    <script defer src="https://cdn.bootcdn.net/ajax/libs/clipboard.js/2.0.6/clipboard.min.js"></script>
    <script defer src="/assets/js/copyCode.js"></script>
    <script defer src="/assets/js/backTop.js"></script>
    <script defer src="/assets/js/tool.js"></script>

    
  <link rel="stylesheet" href="/assets/css/page.css" />
  <link rel="stylesheet" href="/assets/css/sidebar.css" />
  <link rel="stylesheet" href="/assets/css/footer.css" />
  <link rel="stylesheet" href="/assets/css/post.css" />
  <script defer src="/assets/js/backHome.js"></script>
  <script defer src="/assets/js/toc.js"></script>
  <script defer src="/assets/js/copyright.js"></script>


    <title>nodejs爬虫--抓取CSDN某用户全部文章</title>
  </head>
  <body>
    <div class="container">
      <aside>
        
  <div class="sidebar">
  <header>梦的博客</header>
  <div class="info">
    <div class="avatar">
      <img src="https://cdn.jsdelivr.net/gh/xmy6364/blog-image/img/pixelartoc_1.png" alt="头像">
    </div>
    <div class="author">xmy6364</div>
    <div class="description">想要成为大佬</div>
    <div class="about">
      <a href="/about">about me.</a>
    </div>
  </div>
  <div class="links">
    <ul>
    
      <li class="links-item">
        <a href="https://github.com/xmy6364" target="_blank">
          <i class="fa fa-github-alt" aria-hidden="true"></i>
        </a>
      </li>
    
      <li class="links-item">
        <a href="tencent://message/?uin=1176281967" target="_blank">
          <i class="fa fa-qq" aria-hidden="true"></i>
        </a>
      </li>
    
    </ul>
  </div>
  <nav>
    <ul>
    
      <li class="nav-item">
        <a href="/archives">
          <span class="nav-item__count">33</span>
          <span class="nav-item__label">归档</span>
        </a>
      </li>
    
      <li class="nav-item">
        <a href="/categories">
          <span class="nav-item__count">2</span>
          <span class="nav-item__label">分类</span>
        </a>
      </li>
    
      <li class="nav-item">
        <a href="/tags">
          <span class="nav-item__count">45</span>
          <span class="nav-item__label">标签</span>
        </a>
      </li>
    
    </ul>
  </nav>
  <div class="catalogue" id="catalogue"></div>
</div>

      </aside>
      <main>
        
  <div class="post">
    <div class="post-front">
      <h1 class="post-front__title">nodejs爬虫--抓取CSDN某用户全部文章</h1>
      <div class="post-front__desc">
        
        <p class="post-front__desc-date">
          <i class="fa fa-calendar" aria-hidden="true"></i>
          2020/02/11 13:30:55
        </p>
        
        
        <p class="post-front__desc-category">
          <i class="fa fa-folder-o" aria-hidden="true"></i>
          <a href="/categories/技术">
            技术
          </a>
        </p>
        
          <div class="post-front__desc-tags">
          
          <a href="/tags/JavaScript">
            <i class="fa fa-tag" aria-hidden="true"></i>
            JavaScript
          </a>
          
          <a href="/tags/Node.js">
            <i class="fa fa-tag" aria-hidden="true"></i>
            Node.js
          </a>
          
          <a href="/tags/爬虫">
            <i class="fa fa-tag" aria-hidden="true"></i>
            爬虫
          </a>
          
          <a href="/tags/Web Crawler">
            <i class="fa fa-tag" aria-hidden="true"></i>
            Web Crawler
          </a>
          
        </div>
      </div>
    </div>
    <div class="post-content">
      <nav id="toc" class="toc"><ol><li><a href="#准备工作">准备工作</a></li><li><a href="#正式开始">正式开始</a><ol><li><a href="#安装依赖">安装依赖</a></li><li><a href="#创建建好目录">创建建好目录</a></li><li><a href="#创建一个http服务器">创建一个Http服务器</a></li><li><a href="#编写csdnjs模块">编写csdn.js模块</a><ol></ol></li></ol></li><li><a href="#完整代码">完整代码</a></li><li><a href="#参考文章">参考文章</a></li></ol></nav><p>最近正在学习node.js，就像搞一些东西来玩玩，于是这个简单的爬虫就诞生了。</p>
<h2 id="准备工作">准备工作</h2>
<ol>
<li>node.js爬虫肯定要先安装<a href="https://nodejs.org/zh-cn/">node.js</a>环境</li>
<li>创建一个文件夹</li>
<li>在该文件夹打开命令行，执行<code>npm init</code>初始化项目</li>
</ol>
<h2 id="正式开始">正式开始</h2>
<h3 id="安装依赖">安装依赖</h3>
<ul>
<li>express 用来搭建一个简单http服务器，也可以使用node原生api</li>
<li>cheerio 相当于node版的jQuery，用来解析页面</li>
<li>superagent 用来请求目标页面</li>
<li>eventproxy 解决同时处理多个页面的问题</li>
</ul>
<p>直接使用<code>npm install express cheerio superagent eventproxy </code>来安装依赖包，当然你也可以用别的方法。</p>
<h3 id="创建建好目录">创建建好目录</h3>
<pre class="hljs"><code><button class="copy-btn" type="button" data-clipboard-action="copy" data-clipboard-target="#copy1622195864638">复制</button>node-spider-csdn
├─ .gitignore 
├─ node_modules 
├─ README.md 
├─ index.js 			项目入口
├─ package-lock.json
├─ package.json
└─ routes
  └─ csdn.js			爬虫主要代码
<b class="name">shell</b></code><span aria-hidden="true" class="line-numbers-rows"><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span></span></pre><textarea style="position: absolute;top: -9999px;left: -9999px;z-index: -9999;" id="copy1622195864638">node-spider-csdn
├─ .gitignore 
├─ node_modules 
├─ README.md 
├─ index.js 			项目入口
├─ package-lock.json
├─ package.json
└─ routes
  └─ csdn.js			爬虫主要代码
</textarea>
<h3 id="创建一个http服务器">创建一个Http服务器</h3>
<p>在<code>index.js</code>文件中，实例化一个<code>express</code>对象，启动一个Http服务</p>
<pre class="hljs"><code><button class="copy-btn" type="button" data-clipboard-action="copy" data-clipboard-target="#copy1622195050731">复制</button><span class="hljs-keyword">const</span> express = <span class="hljs-built_in">require</span>(<span class="hljs-string">&#x27;express&#x27;</span>);

<span class="hljs-keyword">const</span> app = express();

app.listen(<span class="hljs-number">3000</span>, <span class="hljs-function"><span class="hljs-keyword">function</span>(<span class="hljs-params"></span>) </span>{
    <span class="hljs-built_in">console</span>.log(<span class="hljs-string">&#x27;running in http://127.0.0.1:3000&#x27;</span>);
});
<b class="name">js</b></code><span aria-hidden="true" class="line-numbers-rows"><span></span><span></span><span></span><span></span><span></span><span></span><span></span></span></pre><textarea style="position: absolute;top: -9999px;left: -9999px;z-index: -9999;" id="copy1622195050731">const express = require('express');

const app = express();

app.listen(3000, function() {
    console.log('running in http://127.0.0.1:3000');
});
</textarea>
<p>这样就启动了一个简单的Http本地服务，执行<code>node index.js</code>后通过<code>http://127.0.0.1:3000</code>就可以访问到这个服务器。有关Express的更多内容可以参考<a href="https://expressjs.com/zh-cn/">官方文档</a>。</p>
<h3 id="编写csdnjs模块">编写<code>csdn.js</code>模块</h3>
<p>先引入<code>csdn.js</code>文件并且添加路由</p>
<pre class="hljs"><code><button class="copy-btn" type="button" data-clipboard-action="copy" data-clipboard-target="#copy1622195943598">复制</button><span class="hljs-keyword">const</span> express = <span class="hljs-built_in">require</span>(<span class="hljs-string">&#x27;express&#x27;</span>);
<span class="hljs-keyword">const</span> csdn = <span class="hljs-built_in">require</span>(<span class="hljs-string">&#x27;./routes/csdn.js&#x27;</span>);

<span class="hljs-keyword">const</span> app = express();

app.use(csdn);

app.listen(<span class="hljs-number">3000</span>, <span class="hljs-function"><span class="hljs-keyword">function</span>(<span class="hljs-params"></span>) </span>{
    <span class="hljs-built_in">console</span>.log(<span class="hljs-string">&#x27;running in http://127.0.0.1:3000&#x27;</span>);
});
<b class="name">js</b></code><span aria-hidden="true" class="line-numbers-rows"><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span></span></pre><textarea style="position: absolute;top: -9999px;left: -9999px;z-index: -9999;" id="copy1622195943598">const express = require('express');
const csdn = require('./routes/csdn.js');

const app = express();

app.use(csdn);

app.listen(3000, function() {
    console.log('running in http://127.0.0.1:3000');
});
</textarea>
<p>然后开始编写<code>csdn.js</code></p>
<h4>整体结构</h4>
<pre class="hljs"><code><button class="copy-btn" type="button" data-clipboard-action="copy" data-clipboard-target="#copy1622202328087">复制</button><span class="hljs-comment">// 引入需要的第三方包</span>
<span class="hljs-keyword">const</span> cheerio = <span class="hljs-built_in">require</span>(<span class="hljs-string">&#x27;cheerio&#x27;</span>);
<span class="hljs-keyword">const</span> superagent = <span class="hljs-built_in">require</span>(<span class="hljs-string">&#x27;superagent&#x27;</span>);
<span class="hljs-keyword">const</span> express = <span class="hljs-built_in">require</span>(<span class="hljs-string">&#x27;express&#x27;</span>);
<span class="hljs-keyword">const</span> eventproxy = <span class="hljs-built_in">require</span>(<span class="hljs-string">&#x27;eventproxy&#x27;</span>);

<span class="hljs-keyword">const</span> router = express.Router(); <span class="hljs-comment">// 挂载路由</span>
<span class="hljs-keyword">const</span> ep = <span class="hljs-keyword">new</span> eventproxy();

router.get(<span class="hljs-string">&#x27;/csdn/:name&#x27;</span>, <span class="hljs-function"><span class="hljs-keyword">function</span>(<span class="hljs-params">req, res</span>) </span>{
    <span class="hljs-keyword">const</span> name = req.params.name; <span class="hljs-comment">// 用户id</span>
    <span class="hljs-comment">// 具体实现...</span>
});

<span class="hljs-comment">// 将router暴露出去</span>
<span class="hljs-built_in">module</span>.exports = router;
<b class="name">js</b></code><span aria-hidden="true" class="line-numbers-rows"><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span></span></pre><textarea style="position: absolute;top: -9999px;left: -9999px;z-index: -9999;" id="copy1622202328087">// 引入需要的第三方包
const cheerio = require('cheerio');
const superagent = require('superagent');
const express = require('express');
const eventproxy = require('eventproxy');

const router = express.Router(); // 挂载路由
const ep = new eventproxy();

router.get('/csdn/:name', function(req, res) {
    const name = req.params.name; // 用户id
    // 具体实现...
});

// 将router暴露出去
module.exports = router;
</textarea>
<h4>分析页面</h4>
<p>整体结构写好后就要开始分析CSDN用户文章页面的HTML了。</p>
<p>随便找一个人的博客，经过观察发现：</p>
<ul>
<li>原创文章的完整url:<code>https://blog.csdn.net/l1028386804/article/list/2?t=1</code></li>
<li>CSDN的文章列表是40篇一页</li>
<li>分页控件是动态生成的，所以无法直接通过HTML解析获得</li>
</ul>
<p><img src="https://cdn.jsdelivr.net/gh/xmy6364/blog-image/img/someoneblog.png" alt=""></p>
<p><img src="https://cdn.jsdelivr.net/gh/xmy6364/blog-image/img/someoneblog-1.png" alt=""></p>
<p>然后我们通过开发者工具查看文章列表结构，可以发现：</p>
<ul>
<li>文章信息都在类名为<code>article-item-box</code>的盒子中</li>
<li>id信息在该盒子的<code>data-articleid</code>属性中</li>
</ul>
<p>还有一些其他的信息都很容易能查到，比如博主原创文章总数值等，可以在以后需要的时候再过来查看。</p>
<p><img src="https://cdn.jsdelivr.net/gh/xmy6364/blog-image/img/someoneblog-2.png" alt=""></p>
<h4>获取所有文章页面</h4>
<p>因为无法直接获得分页信息，所以我们通过<code>文章总数 / 每页文章数</code>来获取所有的页面。</p>
<p>首先获取文章的总数：</p>
<pre class="hljs"><code><button class="copy-btn" type="button" data-clipboard-action="copy" data-clipboard-target="#copy1622200494853">复制</button><span class="hljs-comment">/**
 * 获取总文章数目
 * <span class="hljs-doctag">@param <span class="hljs-type">{String}</span> </span>url 页面路径
 * <span class="hljs-doctag">@param <span class="hljs-type">{Function}</span> </span>callback 回调
 */</span>
<span class="hljs-keyword">let</span> getArticleNum = <span class="hljs-function"><span class="hljs-keyword">function</span> (<span class="hljs-params">url, callback</span>) </span>{
    superagent.get(url).end(<span class="hljs-function"><span class="hljs-keyword">function</span> (<span class="hljs-params">err, html</span>) </span>{
        <span class="hljs-keyword">if</span> (err) {
            <span class="hljs-built_in">console</span>.log(<span class="hljs-string">`err = <span class="hljs-subst">${err}</span>`</span>);
        }
        <span class="hljs-keyword">let</span> $ = cheerio.load(html.text);
        <span class="hljs-keyword">let</span> num = <span class="hljs-built_in">parseInt</span>($(<span class="hljs-string">&#x27;.data-info dl&#x27;</span>).first().attr(<span class="hljs-string">&#x27;title&#x27;</span>));

        callback(num);
    });
};
<b class="name">js</b></code><span aria-hidden="true" class="line-numbers-rows"><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span></span></pre><textarea style="position: absolute;top: -9999px;left: -9999px;z-index: -9999;" id="copy1622200494853">/**
 * 获取总文章数目
 * @param {String} url 页面路径
 * @param {Function} callback 回调
 */
let getArticleNum = function (url, callback) {
    superagent.get(url).end(function (err, html) {
        if (err) {
            console.log(`err = ${err}`);
        }
        let $ = cheerio.load(html.text);
        let num = parseInt($('.data-info dl').first().attr('title'));

        callback(num);
    });
};
</textarea>
<p>然后利用简单的循环获取所有文章页面：</p>
<pre class="hljs"><code><button class="copy-btn" type="button" data-clipboard-action="copy" data-clipboard-target="#copy1622201628805">复制</button><span class="hljs-comment">// ...</span>
router.get(<span class="hljs-string">&#x27;/csdn/:name&#x27;</span>, <span class="hljs-function"><span class="hljs-keyword">function</span>(<span class="hljs-params">req, res</span>) </span>{
    <span class="hljs-keyword">const</span> name = req.params.name;
    getArticleNum(<span class="hljs-string">`https://blog.csdn.net/<span class="hljs-subst">${name}</span>`</span>, <span class="hljs-function"><span class="hljs-keyword">function</span> (<span class="hljs-params">num</span>) </span>{
        <span class="hljs-keyword">let</span> pages = []; <span class="hljs-comment">// 保存要抓取的页面</span>

        <span class="hljs-keyword">let</span> pageNum = <span class="hljs-built_in">Math</span>.ceil(num / <span class="hljs-number">40</span>); <span class="hljs-comment">// 计算一共有多少页面</span>

        <span class="hljs-keyword">for</span> (<span class="hljs-keyword">let</span> i = <span class="hljs-number">1</span>; i &lt;= pageNum; i++) {
            pages.push(<span class="hljs-string">`https://blog.csdn.net/<span class="hljs-subst">${name}</span>/article/list/<span class="hljs-subst">${i}</span>?t=1`</span>);
        }
        <span class="hljs-comment">// ...</span>
    });
});
<span class="hljs-comment">// ...</span>
<b class="name">js</b></code><span aria-hidden="true" class="line-numbers-rows"><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span></span></pre><textarea style="position: absolute;top: -9999px;left: -9999px;z-index: -9999;" id="copy1622201628805">// ...
router.get('/csdn/:name', function(req, res) {
    const name = req.params.name;
    getArticleNum(`https://blog.csdn.net/${name}`, function (num) {
        let pages = []; // 保存要抓取的页面

        let pageNum = Math.ceil(num / 40); // 计算一共有多少页面

        for (let i = 1; i <= pageNum; i++) {
            pages.push(`https://blog.csdn.net/${name}/article/list/${i}?t=1`);
        }
        // ...
    });
});
// ...
</textarea>
<p>我们可以通过<code>console.log()</code>或者<code>res.send()</code>来查看获取的网址是否正确</p>
<h4>遍历获取所有页面的HTML</h4>
<pre class="hljs"><code><button class="copy-btn" type="button" data-clipboard-action="copy" data-clipboard-target="#copy1622202439682">复制</button><span class="hljs-comment">// ...</span>
router.get(<span class="hljs-string">&#x27;/csdn/:name&#x27;</span>, <span class="hljs-function"><span class="hljs-keyword">function</span> (<span class="hljs-params">req, res</span>) </span>{
    <span class="hljs-keyword">const</span> name = req.params.name;

    getArticleNum(<span class="hljs-string">`https://blog.csdn.net/<span class="hljs-subst">${name}</span>`</span>, <span class="hljs-function"><span class="hljs-keyword">function</span> (<span class="hljs-params">num</span>) </span>{
        <span class="hljs-keyword">let</span> pages = [];
        <span class="hljs-keyword">let</span> articleData = []; <span class="hljs-comment">// 保存所有文章数据</span>
        
        <span class="hljs-keyword">let</span> pageNum = <span class="hljs-built_in">Math</span>.ceil(num / <span class="hljs-number">40</span>); <span class="hljs-comment">// 计算一共有多少页面</span>

        <span class="hljs-keyword">for</span> (<span class="hljs-keyword">let</span> i = <span class="hljs-number">1</span>; i &lt;= pageNum; i++) {
            pages.push(<span class="hljs-string">`https://blog.csdn.net/<span class="hljs-subst">${name}</span>/article/list/<span class="hljs-subst">${i}</span>?t=1`</span>);
        }

        <span class="hljs-comment">// 获取所有页面的文章信息</span>
        pages.forEach(<span class="hljs-function"><span class="hljs-keyword">function</span> (<span class="hljs-params">targetUrl</span>) </span>{
            superagent.get(targetUrl).end(<span class="hljs-function"><span class="hljs-keyword">function</span> (<span class="hljs-params">err, html</span>) </span>{
                <span class="hljs-keyword">if</span> (err) {
                    <span class="hljs-built_in">console</span>.log(<span class="hljs-string">`err <span class="hljs-subst">${err}</span>`</span>);
                }
                <span class="hljs-keyword">let</span> $ = cheerio.load(html.text);
                
				<span class="hljs-comment">// 当前页面的文章列表</span>
                <span class="hljs-keyword">let</span> articlesHtml = $(<span class="hljs-string">&#x27;.article-list .article-item-box&#x27;</span>);

                <span class="hljs-comment">// 遍历当前页的文章列表</span>
                <span class="hljs-keyword">for</span> (<span class="hljs-keyword">let</span> i = <span class="hljs-number">0</span>; i &lt; articlesHtml.length; i++) {
                    <span class="hljs-comment">// 解析获取文章信息</span>
                    <span class="hljs-comment">// push到articleData中</span>
                    <span class="hljs-comment">// ...</span>
                }
            });
        });
    });
});
<span class="hljs-comment">// ...</span>
<b class="name">js</b></code><span aria-hidden="true" class="line-numbers-rows"><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span></span></pre><textarea style="position: absolute;top: -9999px;left: -9999px;z-index: -9999;" id="copy1622202439682">// ...
router.get('/csdn/:name', function (req, res) {
    const name = req.params.name;

    getArticleNum(`https://blog.csdn.net/${name}`, function (num) {
        let pages = [];
        let articleData = []; // 保存所有文章数据
        
        let pageNum = Math.ceil(num / 40); // 计算一共有多少页面

        for (let i = 1; i <= pageNum; i++) {
            pages.push(`https://blog.csdn.net/${name}/article/list/${i}?t=1`);
        }

        // 获取所有页面的文章信息
        pages.forEach(function (targetUrl) {
            superagent.get(targetUrl).end(function (err, html) {
                if (err) {
                    console.log(`err ${err}`);
                }
                let $ = cheerio.load(html.text);
                
				// 当前页面的文章列表
                let articlesHtml = $('.article-list .article-item-box');

                // 遍历当前页的文章列表
                for (let i = 0; i < articlesHtml.length; i++) {
                    // 解析获取文章信息
                    // push到articleData中
                    // ...
                }
            });
        });
    });
});
// ...
</textarea>
<h4>解析文章信息</h4>
<p>因为获取到的有些文本中空格太多，所以需要用到正则表达式来去除多余的空格。</p>
<p><code>cheerio</code>对于Document的操作和<code>jQuery</code>基本一样，所以有前端基础的可以很轻松上手。</p>
<pre class="hljs"><code><button class="copy-btn" type="button" data-clipboard-action="copy" data-clipboard-target="#copy1622199486094">复制</button><span class="hljs-comment">/**
 * 解析html字符串，获取文章信息
 * <span class="hljs-doctag">@param <span class="hljs-type">{String}</span> </span>html 包含文章信息的html
 * <span class="hljs-doctag">@param <span class="hljs-type">{Number}</span> </span>index 文章索引
 */</span>
<span class="hljs-keyword">let</span> analysisHtml = <span class="hljs-function"><span class="hljs-keyword">function</span> (<span class="hljs-params">html, index</span>) </span>{
    <span class="hljs-keyword">return</span> {
        <span class="hljs-attr">id</span>: html.eq(index).attr(<span class="hljs-string">&#x27;data-articleid&#x27;</span>),
        <span class="hljs-attr">title</span>: html.eq(index).find(<span class="hljs-string">&#x27;h4 a&#x27;</span>).text().replace(<span class="hljs-regexp">/\s+/g</span>, <span class="hljs-string">&#x27;&#x27;</span>).slice(<span class="hljs-number">2</span>),
        <span class="hljs-attr">link</span>: html.eq(index).find(<span class="hljs-string">&#x27;a&#x27;</span>).attr(<span class="hljs-string">&#x27;href&#x27;</span>),
        <span class="hljs-attr">abstract</span>: html.eq(index).find(<span class="hljs-string">&#x27;.content a&#x27;</span>).text().replace(<span class="hljs-regexp">/\s+/g</span>, <span class="hljs-string">&#x27;&#x27;</span>),
        <span class="hljs-attr">shared_time</span>: html.eq(index).find(<span class="hljs-string">&#x27;.info-box .date&#x27;</span>).text().replace(<span class="hljs-regexp">/\s+/</span>, <span class="hljs-string">&#x27;&#x27;</span>),
        <span class="hljs-attr">read_count</span>: html.eq(index).find(<span class="hljs-string">&#x27;.info-box .read-num .num&#x27;</span>).first().text().replace(<span class="hljs-regexp">/\s+/</span>, <span class="hljs-string">&#x27;&#x27;</span>),
        <span class="hljs-attr">comment_count</span>: html.eq(index).find(<span class="hljs-string">&#x27;.info-box .read-num .num&#x27;</span>).last().text().replace(<span class="hljs-regexp">/\s+/</span>, <span class="hljs-string">&#x27;&#x27;</span>)
    };
};
<b class="name">js</b></code><span aria-hidden="true" class="line-numbers-rows"><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span></span></pre><textarea style="position: absolute;top: -9999px;left: -9999px;z-index: -9999;" id="copy1622199486094">/**
 * 解析html字符串，获取文章信息
 * @param {String} html 包含文章信息的html
 * @param {Number} index 文章索引
 */
let analysisHtml = function (html, index) {
    return {
        id: html.eq(index).attr('data-articleid'),
        title: html.eq(index).find('h4 a').text().replace(/\s+/g, '').slice(2),
        link: html.eq(index).find('a').attr('href'),
        abstract: html.eq(index).find('.content a').text().replace(/\s+/g, ''),
        shared_time: html.eq(index).find('.info-box .date').text().replace(/\s+/, ''),
        read_count: html.eq(index).find('.info-box .read-num .num').first().text().replace(/\s+/, ''),
        comment_count: html.eq(index).find('.info-box .read-num .num').last().text().replace(/\s+/, '')
    };
};
</textarea>
<pre class="hljs"><code><button class="copy-btn" type="button" data-clipboard-action="copy" data-clipboard-target="#copy1622200173225">复制</button><span class="hljs-comment">// ...</span>
<span class="hljs-comment">// 遍历当前页的文章列表</span>
<span class="hljs-keyword">for</span> (<span class="hljs-keyword">let</span> i = <span class="hljs-number">0</span>; i &lt; articlesHtml.length; i++) {
    <span class="hljs-keyword">let</span> article = analysisHtml(articlesHtml, i);
    articleData.push(article);
    <span class="hljs-comment">// ...</span>
}
<span class="hljs-comment">// ...</span>
<b class="name">js</b></code><span aria-hidden="true" class="line-numbers-rows"><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span></span></pre><textarea style="position: absolute;top: -9999px;left: -9999px;z-index: -9999;" id="copy1622200173225">// ...
// 遍历当前页的文章列表
for (let i = 0; i < articlesHtml.length; i++) {
    let article = analysisHtml(articlesHtml, i);
    articleData.push(article);
    // ...
}
// ...
</textarea>
<p>我们已经获取到所有文章的信息数据，但是因为获取各个页面的文章时是并发异步进行的，所以要同时利用这些数据特殊的方法。</p>
<h4>处理并发异步操作</h4>
<p>这里我使用的是“计数器”<code>eventproxy</code>，还有很多其他的方法都可以解决这个问题。</p>
<pre class="hljs"><code><button class="copy-btn" type="button" data-clipboard-action="copy" data-clipboard-target="#copy1622199898830">复制</button><span class="hljs-comment">// ...</span>
pages.forEach(<span class="hljs-function"><span class="hljs-keyword">function</span> (<span class="hljs-params">targetUrl</span>) </span>{
    superagent.get(targetUrl).end(<span class="hljs-function"><span class="hljs-keyword">function</span> (<span class="hljs-params">err, html</span>) </span>{
        <span class="hljs-keyword">if</span> (err) {
            <span class="hljs-built_in">console</span>.log(<span class="hljs-string">`err <span class="hljs-subst">${err}</span>`</span>);
        }
        <span class="hljs-keyword">let</span> $ = cheerio.load(html.text);

        <span class="hljs-keyword">let</span> articlesHtml = $(<span class="hljs-string">&#x27;.article-list .article-item-box&#x27;</span>);

        <span class="hljs-keyword">for</span> (<span class="hljs-keyword">let</span> i = <span class="hljs-number">0</span>; i &lt; articlesHtml.length; i++) {
            <span class="hljs-keyword">let</span> article = analysisHtml(articlesHtml, i);
            articleData.push(article);

            ep.emit(<span class="hljs-string">&#x27;blogArtc&#x27;</span>, article); <span class="hljs-comment">// 计数器</span>
        }
    });
});

<span class="hljs-comment">// 当所有&#x27;blogArtc&#x27;完成后，触发回调</span>
ep.after(<span class="hljs-string">&#x27;blogArtc&#x27;</span>, num, <span class="hljs-function"><span class="hljs-keyword">function</span> (<span class="hljs-params">data</span>) </span>{
    res.json({
        <span class="hljs-attr">status_code</span>: <span class="hljs-number">0</span>,
        <span class="hljs-attr">data</span>: data
    });
});
<span class="hljs-comment">// ...</span>
<b class="name">js</b></code><span aria-hidden="true" class="line-numbers-rows"><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span><span></span></span></pre><textarea style="position: absolute;top: -9999px;left: -9999px;z-index: -9999;" id="copy1622199898830">// ...
pages.forEach(function (targetUrl) {
    superagent.get(targetUrl).end(function (err, html) {
        if (err) {
            console.log(`err ${err}`);
        }
        let $ = cheerio.load(html.text);

        let articlesHtml = $('.article-list .article-item-box');

        for (let i = 0; i < articlesHtml.length; i++) {
            let article = analysisHtml(articlesHtml, i);
            articleData.push(article);

            ep.emit('blogArtc', article); // 计数器
        }
    });
});

// 当所有'blogArtc'完成后，触发回调
ep.after('blogArtc', num, function (data) {
    res.json({
        status_code: 0,
        data: data
    });
});
// ...
</textarea>
<p>这样，一个简单的node爬虫就写好了，执行<code>node index.js</code>启动服务后，在浏览器中输入<code>http://127.0.0.1:3000/csdn/xxxx</code>就可以获得xxxx（这是id）的全部文章了。</p>
<h2 id="完整代码">完整代码</h2>
<ul>
<li><a href="https://github.com/xmy6364/node-spider-csdn">node-spider-csdn</a></li>
</ul>
<h2 id="参考文章">参考文章</h2>
<ul>
<li>
<p><a href="https://juejin.im/post/5b4f007fe51d4519277b9707#heading-1">分分钟教你用node.js写个爬虫</a></p>
</li>
<li>
<p><a href="https://www.cnblogs.com/coco1s/p/4954063.html">【nodeJS爬虫】前端爬虫系列 -- 小爬「博客园」</a></p>
</li>
<li>
<p><a href="https://github.com/jiayisheji/blog/issues/7">10分钟教你撸一个nodejs爬虫系统</a></p>
</li>
<li>
<p><a href="https://www.jianshu.com/p/a86bd4bd48c9">node.js 学习笔记004:使用eventproxy控制并发</a></p>
</li>
</ul>

    </div>
    
  </div>

        <footer>
        
  <div class="footer">
  
  <div class="theme">
    博客主题为 <a href="https://github.com/xmy6364/CoinRailgun">CoinRailgun</a> 默认主题
  </div>
  <div class="copyright">
    <span>© 2019-2021 xmy6364.</span>
    <span>Powered by <a href="https://github.com/xmy6364/CoinRailgun">CoinRailgun</a></span>
  </div>
</div>

        </footer>
      </main>
    </div>
  </body>
</html>
